{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "[Learn the Basics](intro.html) ||\n",
        "[Quickstart](quickstart_tutorial.html) ||\n",
        "[Tensors](tensorqs_tutorial.html) ||\n",
        "[Datasets & DataLoaders](data_tutorial.html) ||\n",
        "[Transforms](transforms_tutorial.html) ||\n",
        "[Build Model](buildmodel_tutorial.html) ||\n",
        "**Autograd** ||\n",
        "[Optimization](optimization_tutorial.html) ||\n",
        "[Save & Load Model](saveloadrun_tutorial.html)\n",
        "\n",
        "# Automatic Differentiation with ``torch.autograd``\n",
        "\n",
        "When training neural networks, the most frequently used algorithm is\n",
        "**back propagation**. In this algorithm, parameters (model weights) are\n",
        "adjusted according to the **gradient** of the loss function with respect\n",
        "to the given parameter.\n",
        "\n",
        "To compute those gradients, PyTorch has a built-in differentiation engine\n",
        "called ``torch.autograd``. It supports automatic computation of gradient for any\n",
        "computational graph.\n",
        "\n",
        "Consider the simplest one-layer neural network, with input ``x``,\n",
        "parameters ``w`` and ``b``, and some loss function. It can be defined in\n",
        "PyTorch in the following manner:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "collapsed": false
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "x =  tensor([1., 1., 1., 1., 1.])\n",
            "y =  tensor([0., 0., 0.])\n",
            "w =  tensor([[ 0.6614,  0.2669,  0.0617],\n",
            "        [ 0.6213, -0.4519, -0.1661],\n",
            "        [-1.5228,  0.3817, -1.0276],\n",
            "        [-0.5631, -0.8923, -0.0583],\n",
            "        [-0.1955, -0.9656,  0.4224]], requires_grad=True)\n",
            "b =  tensor([ 0.2673, -0.4212, -0.5107], requires_grad=True)\n",
            "z =  tensor([-0.7313, -2.0824, -1.2786], grad_fn=<AddBackward0>)\n",
            "tensor(0.2520, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "\n",
        "torch.manual_seed(1)\n",
        "\n",
        "x = torch.ones(5)  # input tensor\n",
        "y = torch.zeros(3)  # expected output\n",
        "w = torch.randn(5, 3, requires_grad=True)\n",
        "b = torch.randn(3, requires_grad=True)\n",
        "z = torch.matmul(x, w)+b\n",
        "loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)\n",
        "\n",
        "print('x = ', x)\n",
        "print('y = ', y)\n",
        "print('w = ', w)\n",
        "print('b = ', b)\n",
        "print('z = ', z)\n",
        "print(loss)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Tensors, Functions and Computational graph\n",
        "\n",
        "This code defines the following **computational graph**:\n",
        "\n",
        "<!-- .. figure:: images/comp-graph.png  :alt: -->\n",
        "\n",
        "![Computational graph](images/comp-graph.png)\n",
        "\n",
        "\n",
        "In this network, ``w`` and ``b`` are **parameters**, which we need to\n",
        "optimize. Thus, we need to be able to compute the gradients of loss\n",
        "function with respect to those variables. In order to do that, we set\n",
        "the ``requires_grad`` property of those tensors.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<div class=\"alert alert-info\"><h4>Note</h4><p>You can set the value of ``requires_grad`` when creating a\n",
        "          tensor, or later by using ``x.requires_grad_(True)`` method.</p></div>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "A function that we apply to tensors to construct computational graph is\n",
        "in fact an object of class `Function`. This object knows how to\n",
        "compute the function in the _forward_ direction, and also how to compute\n",
        "its derivative during the _backward propagation_ step. A reference to\n",
        "the backward propagation function is stored in `grad_fn` property of a\n",
        "tensor. You can find more information of `Function` [in the\n",
        "documentation](https://pytorch.org/docs/stable/autograd.html#function)\\_.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "collapsed": false
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Gradient function for z = <AddBackward0 object at 0x0000018F3320FCA0>\n",
            "Gradient function for loss = <BinaryCrossEntropyWithLogitsBackward0 object at 0x0000018F3320FF40>\n"
          ]
        }
      ],
      "source": [
        "print(f\"Gradient function for z = {z.grad_fn}\")\n",
        "print(f\"Gradient function for loss = {loss.grad_fn}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Computing Gradients\n",
        "\n",
        "To optimize weights of parameters in the neural network, we need to\n",
        "compute the derivatives of our loss function with respect to parameters,\n",
        "namely, we need $\\frac{\\partial loss}{\\partial w}$ and\n",
        "$\\frac{\\partial loss}{\\partial b}$ under some fixed values of\n",
        "``x`` and ``y``. To compute those derivatives, we call\n",
        "``loss.backward()``, and then retrieve the values from ``w.grad`` and\n",
        "``b.grad``:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "collapsed": false
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[0.1083, 0.0369, 0.0726],\n",
            "        [0.1083, 0.0369, 0.0726],\n",
            "        [0.1083, 0.0369, 0.0726],\n",
            "        [0.1083, 0.0369, 0.0726],\n",
            "        [0.1083, 0.0369, 0.0726]])\n",
            "tensor([0.1083, 0.0369, 0.0726])\n"
          ]
        }
      ],
      "source": [
        "loss.backward()\n",
        "print(w.grad)\n",
        "print(b.grad)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<div class=\"alert alert-info\"><h4>Note</h4>\n",
        "\n",
        "- We can only obtain the ``grad`` properties for the leaf nodes of the computational graph, which have ``requires_grad`` \n",
        "property  set to ``True``. For all other nodes in our graph, gradients will not be  available.\n",
        "- We can only perform gradient calculations using ``backward`` once on a given graph, for performance reasons. If we need to do several ``backward`` calls on the same graph, we need to pass  ``retain_graph=True`` to the ``backward`` call.\n",
        "\n",
        "</div>\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Disabling Gradient Tracking\n",
        "\n",
        "By default, all tensors with ``requires_grad=True`` are tracking their\n",
        "computational history and support gradient computation. However, there\n",
        "are some cases when we do not need to do that, for example, when we have\n",
        "trained the model and just want to apply it to some input data, i.e. we\n",
        "only want to do *forward* computations through the network. We can stop\n",
        "tracking computations by surrounding our computation code with\n",
        "``torch.no_grad()`` block:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "collapsed": false
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "True\n",
            "False\n"
          ]
        }
      ],
      "source": [
        "z = torch.matmul(x, w)+b\n",
        "print(z.requires_grad)\n",
        "\n",
        "with torch.no_grad():\n",
        "    z = torch.matmul(x, w)+b\n",
        "print(z.requires_grad)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Another way to achieve the same result is to use the ``detach()`` method\n",
        "on the tensor:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "collapsed": false
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "False\n"
          ]
        }
      ],
      "source": [
        "z = torch.matmul(x, w)+b\n",
        "z_det = z.detach()\n",
        "print(z_det.requires_grad)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "There are reasons you might want to disable gradient tracking:\n",
        "  - To mark some parameters in your neural network as **frozen parameters**. This is\n",
        "    a very common scenario for\n",
        "    [finetuning a pretrained network](https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html)_\n",
        "  - To **speed up computations** when you are only doing forward pass, because computations on tensors that do\n",
        "    not track gradients would be more efficient.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## More on Computational Graphs\n",
        "Conceptually, autograd keeps a record of data (tensors) and all executed\n",
        "operations (along with the resulting new tensors) in a directed acyclic\n",
        "graph (DAG) consisting of\n",
        "[Function](https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)_\n",
        "objects. In this DAG, leaves are the input tensors, roots are the output\n",
        "tensors. By tracing this graph from roots to leaves, you can\n",
        "automatically compute the gradients using the chain rule.\n",
        "\n",
        "In a forward pass, autograd does two things simultaneously:\n",
        "\n",
        "- run the requested operation to compute a resulting tensor\n",
        "- maintain the operation’s *gradient function* in the DAG.\n",
        "\n",
        "The backward pass kicks off when ``.backward()`` is called on the DAG\n",
        "root. ``autograd`` then:\n",
        "\n",
        "- computes the gradients from each ``.grad_fn``,\n",
        "- accumulates them in the respective tensor’s ``.grad`` attribute\n",
        "- using the chain rule, propagates all the way to the leaf tensors.\n",
        "\n",
        "<div class=\"alert alert-info\"><h4>Note</h4><p>**DAGs are dynamic in PyTorch**\n",
        "  An important thing to note is that the graph is recreated from scratch; after each\n",
        "  ``.backward()`` call, autograd starts populating a new graph. This is\n",
        "  exactly what allows you to use control flow statements in your model;\n",
        "  you can change the shape, size and operations at every iteration if\n",
        "  needed.</p></div>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Optional Reading: Tensor Gradients and Jacobian Products\n",
        "\n",
        "In many cases, we have a scalar loss function, and we need to compute\n",
        "the gradient with respect to some parameters. However, there are cases\n",
        "when the output function is an arbitrary tensor. In this case, PyTorch\n",
        "allows you to compute so-called **Jacobian product**, and not the actual\n",
        "gradient.\n",
        "\n",
        "For a vector function $\\vec{y}=f(\\vec{x})$, where\n",
        "$\\vec{x}=\\langle x_1,\\dots,x_n\\rangle$ and\n",
        "$\\vec{y}=\\langle y_1,\\dots,y_m\\rangle$, a gradient of\n",
        "$\\vec{y}$ with respect to $\\vec{x}$ is given by **Jacobian\n",
        "matrix**:\n",
        "\n",
        "\\begin{align}J=\\left(\\begin{array}{ccc}\n",
        "      \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{1}}{\\partial x_{n}}\\\\\n",
        "      \\vdots & \\ddots & \\vdots\\\\\n",
        "      \\frac{\\partial y_{m}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n",
        "      \\end{array}\\right)\\end{align}\n",
        "\n",
        "Instead of computing the Jacobian matrix itself, PyTorch allows you to\n",
        "compute **Jacobian Product** $v^T\\cdot J$ for a given input vector\n",
        "$v=(v_1 \\dots v_m)$. This is achieved by calling ``backward`` with\n",
        "$v$ as an argument. The size of $v$ should be the same as\n",
        "the size of the original tensor, with respect to which we want to\n",
        "compute the product:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "collapsed": false
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "First call\n",
            "tensor([[4., 2., 2., 2., 2.],\n",
            "        [2., 4., 2., 2., 2.],\n",
            "        [2., 2., 4., 2., 2.],\n",
            "        [2., 2., 2., 4., 2.]])\n",
            "\n",
            "Second call\n",
            "tensor([[8., 4., 4., 4., 4.],\n",
            "        [4., 8., 4., 4., 4.],\n",
            "        [4., 4., 8., 4., 4.],\n",
            "        [4., 4., 4., 8., 4.]])\n",
            "\n",
            "Call after zeroing gradients\n",
            "tensor([[4., 2., 2., 2., 2.],\n",
            "        [2., 4., 2., 2., 2.],\n",
            "        [2., 2., 4., 2., 2.],\n",
            "        [2., 2., 2., 4., 2.]])\n"
          ]
        }
      ],
      "source": [
        "inp = torch.eye(4, 5, requires_grad=True)\n",
        "out = (inp+1).pow(2).t()\n",
        "out.backward(torch.ones_like(out), retain_graph=True)\n",
        "print(f\"First call\\n{inp.grad}\")\n",
        "out.backward(torch.ones_like(out), retain_graph=True)\n",
        "print(f\"\\nSecond call\\n{inp.grad}\")\n",
        "inp.grad.zero_()\n",
        "out.backward(torch.ones_like(out), retain_graph=True)\n",
        "print(f\"\\nCall after zeroing gradients\\n{inp.grad}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Notice that when we call ``backward`` for the second time with the same\n",
        "argument, the value of the gradient is different. This happens because\n",
        "when doing ``backward`` propagation, PyTorch **accumulates the\n",
        "gradients**, i.e. the value of computed gradients is added to the\n",
        "``grad`` property of all leaf nodes of computational graph. If you want\n",
        "to compute the proper gradients, you need to zero out the ``grad``\n",
        "property before. In real-life training an *optimizer* helps us to do\n",
        "this.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<div class=\"alert alert-info\"><h4>Note</h4><p>Previously we were calling ``backward()`` function without\n",
        "          parameters. This is essentially equivalent to calling\n",
        "          ``backward(torch.tensor(1.0))``, which is a useful way to compute the\n",
        "          gradients in case of a scalar-valued function, such as loss during\n",
        "          neural network training.</p></div>\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "--------------\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Further Reading\n",
        "- [Autograd Mechanics](https://pytorch.org/docs/stable/notes/autograd.html)\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.9"
    },
    "vscode": {
      "interpreter": {
        "hash": "cf18841ace8313d0bc088ca146c17a6c0040e82121d5cb75c0ea07172309253d"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
